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(54) Method and apparatus for annotation of medical imagery to facilitate patient identification, 
diagnosis and treatment 

(57) A method and apparatus for annotation of med- 
ical imagery to facilitate patient identification, diagnosis, 
and treatment is characterized by an imaging device for 
producing a first signal representative of sensed char- 
. acteristics of the individual and a minutiae generator 
which receives the first signal and produces a second 
signal representative of minutiae of the individual, the 
minutiae corresponding to specific branch points of 
blood vessels of the individual A minutiae data genera- 
tor analyzes the characteristics of min utiae and produc- 
es a third signal representative of the characteristics 
which is stored in a minutiae database for each of the 
plurality of known individuals and their medical condi- 
tions. The minutiae and minutiae data may be used to 
annotate medical imagery to facilitate subsequent im- 
age comparison by providing standardized registration 
points and time-varying characteristics. A minutiae 
matcher pairs corresponding second signals and third 
signals from a current patient with those from a data- 
base record, and the paired signals are used to align the 
images and compare them. The minutiae analysis tech- 
niques of the invention can be used to identify medical 
patients, assist in ihe diagnosis of medical conditions, 
and detect and monitor the use of alcohol and drugs, 
including anesthesia. 
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quality than the rolled prints, since inconsistencies in the 
amount of ink applied and in the pressure used to trans- 
fer the print to paper are not a factor. 

Automated fingerprint matching techniques have 
been developed which rapidly classify an unknown print 
and then search through the portion of the database as- 
sociated with that class looking for a match. Unknown 
prints. may be from a "tenprint" card, or may be latent 
prints which have been lifted from a crime scene. A la- 
tent print may include a sizable area of one or more fin- 
gers, such as on a water glass, or it may include only a 
portion of one or more fingers, such as on a telephone 
keypad. Latent prints may be found on top of other latent 
prints, such as when several people have used the 
same telephone. 

Matching techniques often extract minutiae points 
from the prints, and then compare the sets of minutiae 
rather than compare entire prints various classifica- 
tions of minutiae types have been proposed by different 
companies and authorities. An example is given from 
the Costello U.S. Patent No. 4,947,443. Six types 
oPcharacteristic features" are presented in this patent, 
each one relating to a type of minutia. This fingerprint 
matching technique references the type, orientation, 
and location of each characteristic and each and every 
other characteristic. Using this approach, on the order 
of 80 to 150 minutia points are identified in each finger- 
print. Other fingerprint minutiae extraction and matching 
patents produce essentially the same number of minu- 
tiae, with difference in what features of the set of minu- 
tiae are considered in attempted matching and in how 
the matching is performed. In U.S. courts, evidentiary 
rules have traditionally required that 16 or more minutia 
points be found to correspond between two prints in or- 
der for them to be considered to be from the same per- 
son. The determination of likely matching prints is gen- 
erally assisted or performed entirely by a computer sys- 
tem; however, the final decree of a match is made by a 
fingerprint expert, who reviews the computer system re- 
sults. 

Matches between different prints taken from the 
same finger are never perfect, since the fingers are de- 
formable, three-dimensional, connected and jointed 
structures which leave two-dimensional prints on sur- 
faces they encounter through pressure. The exact an- 
gles between the fingers and the surfaces, the amount 
and direction of pressure, and the effect of movement 
between the fingers and the surfaces all cause varia- 
tions in the exact prints produced. Even when prints are 
produced by a live scan technique variation in the light- 
ing, hand position, oil or dust on the fingers, use ol lo- 
tions, and scratches or paper cuts will produce minor 
variations in the prints produced. 

Therefore, the exact number position, and charac- 
teristics of minutiae extracted from two prints may be 
different even though they are produced by the same 
finger. The challenge for an automated fingerprint iden- 
tification system is to recognize allowable minor varia- 



tions in actual matching prints while not allowing varia- 
tions so wide that mismatches occur. Several AFIS 
products are now commercially offered which provide 
acceptable accuracy. Local and regional police forces 

5 may use smaller databases which contain only the prints 
of persons historically associated with their areas, rather 
than relying on federal resources to search the entire 
nationwide FBI files. Smaller scale fingerprint system, 
such as those associated with a system which controls 

io access to an office building, may use the same minutiae 
matching techniques. 

With rolled and live-scan prints, the orientation 
ofeach print, and the finger to which it corresponds is 
know Also, quality checks can be built into the process 

is such that repeat prints may be taken to insure quality 
when needed. In the case of latenis, however, the anal- 
ysis is done after the fact. It is not known which finger 
left the print, and the orientation of the finger may be in 
doubt when only a partial print is found. Therefore, 

20 matching of latents is much more difficult than matching 
of rolled or live scan prints. 

Various minutiae extraction algorithms are used in 
current fingerprint identification systems, some of which 
merely utilize the location of the minutia points and oth- 

25 ers of which utilize also additional information about the 
type of minutia each point represents. For example, sim- 
ple graph matching techniques can be used to compare 
the follow-lhe-dots vectors generated by connecting the 
minutia points in order forced by considering intersec- 

30 tions with a spiral from the centerpoint of the fingerprint. 
Alternately, the ridge angle at each minutia point can be 
considered and matched along with the coordinates, in 
a best-fit attempt to match each unknown print to each 
known print. A measure of goodness of fit can then be 

35 computed and used to rank other possible matches. 

U S. patent No. 4,525,859 to Bowles teaches a pat- 
tern recognition system which detects line bifurcations 
and line endings, denoted minutiae, in a pattern of lines 
such as are found in a fingerprint. According to this ref- 

40 erence, the FBI uses an automatic fingerprint identifica- 
tion system entitled "FINDER" which uses an optical 
scan reader. The information is then enhanced to elim- 
inate grays and fill in gaps in the ridges. A 16x16 incre- 
ment square window scans the fingerprint, an increment 

45 being a tenth of a millimeter Thus, a window advances 
through the fingerprint in increments of tenth of millim- 
eter and looks for ridges which enter the window but do 
not exit it. When such a ridge is identified, its coordinate 
location is stored and the ridge is analyzed to establish 

so an angle, theta, of the ridge at the termination. The data 
are then re-scanned to look for terminations of valleys, 
which are ridge bifurcations. The additional coordinates 
and angles of each of the inverted ending points also 
are stored. 

55 in latent prints, the distances between ridges of a 
fingerprint average 0.4 millimeters but can vary by a fac- 
tor of 2 for any individual finger depending on skin dis- 
placement when the finger contacts the hard surface 
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are compared by computer to the set of vectors of 
known prints: and 

Sixth, the comparison resultsare used lo select po- 
tential matches and provide a goodness of fit indi- 
cation between the unknown and known prints. 

Numerous approaches to recognition using visible 
light imaging of faces have been proposed. Many of 
them apply standard pattern matching techniques: oth- 
ers involve definition of face metrics. 

U.S. patent No. 4,975.969 to Tal discloses a method 
and apparatus for uniquely identifying individuals by 
measurement of particular physical characteristics 
viewable by the naked eye or by imaging in the visible 
spectrum. This reference defined facial parameters 
which are the distances between identifiable parame- 
ters on the human face, and/or ratios of the facial pa- 
rameters, and teaches that they can be used to identify 
an individual since the set of parameters for each indi- 
vidual is unique. 

Tal's approach utilizes visible features on the face, 
and therefore cannot be relied upon to distinguish be- 
tween faces having similar visual features, for example 
as would be the case with identical twins. In addition, 
the "rubber sheeting" effect caused by changes in facial 
expression, the aging effects which cause lengthening 
of the nose, thinning of the lips, wrinkles, and deepening 
of the creases on the sides of the nose, all cause chang- 
es in the parameters and in ratios relied on in this meth- 
od. Furthermore, the parameters and ratios of any par- 
ticular person's face may be measured by anyone taking 
a photograph, and thereby used to select or disguise 
another person to appear to be that person. Therefore, 
the security provided by such a technique may not be 
adequate for unattended or highly sensitive locations 

Still another known scheme utilizes eigenanalysis 
of visual face images to develop a set of characteristic 
features Pentland, View-Based and Modular Ei- 
genspaces for Face Recognition, MIT Media Laboratory 
Perceptual Computing Section, Technical Report No. 
245. Faces are then described in terms of weighting on 
those features. The approach claims to accommodate 
head position changes and the wearing of glasses, as 
well as changes in facial expressions. This disclosure • 
teaches that pre-processing for registration is essential 
to eigenvector recognition systems. The processing re- 
quired to establish the eigenvector set is extensive, es- 
pecially for large databases. Addition of new faces to 



caused by facial expressions, makeup, sunburns, shad- 
ows and similar unintentional disguises. Detecting the 
wearing of disguises and distinguishing between iden- 
tical twins may be done from visible imagery if sufficient 

5 resolution and controlled lighting is available. However, 
that significantly increases the computational complex- 
ity of the identification task, and makes the recognition 
accuracy vulnerable to unintentional normal variations. 
From the standpoint of evidentiary use, it might also 

io be argued that the application of eigenanalysis to a very 
large database of faces, such as all mug shots in the 
FBI files, would be considered so esoteric by the public 
at large that automated matches based upon its use will 
not readily be acceptable to a jury as convincing evi- 

'5 dence of identity. By comparison, techniques based on 
minutiae matching technique, such as are used with fin- 
gerprint identification, would be expected to find a more 
understanding reception by the law enforcement com- 
munity, and to be more acceptable for evidentiary pur- 

20 poses within a reasonable number of years after their 
introduction. 

One known scheme using facial thermograms for 
identification is described in the Prokoski et al U.S. Pat- 
ent No. 5,163,094 which discloses defining "elemental 

25 shapes" in the surface thermal image produced by the 
underlying vascular structure of blood vessels beneath 
the skin. Depending on the environment of use, thermal 
facial identification may provide greater security over 
identilication from visual images and may therefore be 

30 considered preferable. It is extremely difficult, if not im- 
possible, to counterfeit or forge one face to look like an- 
other in infrared, whereas it is often possible to disguise 
one person to look like another in visible light. However, 
the use of elemental shapes is found in practice to be 

35 vulnerable to such variables as head rotation and tilt, 
ambient and physiological temperature changes, varia- 
tions in imaging and processing systems, and distor- 
tions or obstructions in a facial image (e.g., due to eye- 
glasses). 

>o Eigenanalysis of the elemental shapes of a thermal 
facial image has also been used for recognition. In one 
approach, several sets of elemental shapes are pro- 
duced for each image by imposing different thermal 
banding constraints. The totality of shapes are then an- 
'S alyzed with respect to a library of facial thermal images 
Eigenshape analysis is used to compare the character- 
istics of shapes in each person's images. Eleven char- 
acteristics of each shape are considered, including: pe- 
rimeter, area, centroid x and y locations, minimum and 



the database requires the re-running of the eigenanal- so maximum chord length through the centroid, standard 



ysis. Accordingly, use of eigenanalysis may not be ap- 
propriate for use in a general face identification system 
such as would be analogous to the FBI's and AFIS fin- 
gerprint system. 

Visible metrics typically require ground truth dis- 
tance measurements unless they rely strictly upon ratios 
of measurements. Thus, such systems can be fooled by 
intentional disguises, and they are subject lo variations 



deviation of that length, minimum and maximum chord 
length between perimeter points, standard deviation of 
that length, and area/perimeter. 

Each person's image is then characterized by a set 
of 1 1 -coefficient vectors. The difference in eigenspace 
between any two images is calculated to yield a meas- 
urement to which a threshold was applied to make 
a"match/no match" decision In practice, such a system 
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Fig 1 3 illustrates the corresponding anatomy for the 
thermogram of Fig. 12; 



Fig. 14 is a block diagram representing apparatus 
for maintaining the position of a surgical instrument 
relative to a surgical site according to the invention; 

Fig. 1 5 is a facial thermogram of an alcohol-free in- 
dividual; 

Fig. 16 is a facial thermogram of the individual of 
Fig. 15 under the influence of alcohol: 

Fig. 17 is a graph representing the thermal signa- 
tures of selected minutia points of an individual prior 
and subsequent to use of alcohol: 

Fig. 18 is a block diagram of the apparatus lor de- 
tection of alcohol and drug use by an individual ac- 
cording to the invention; 

Fig. 1 9 is a graph representing the results of a sta- 
tistical analysis of drug users in a random popula- 
tion; and 

Fig. 20 is a block diagram of the apparatus for de- 
tection of alcohol and drug use in a random popu- 
lation. 

DETAILED DESCRIPTION 

Facial Minutiae Extraction 

In Fig. 1 , there is shown a system 1 00 for personal 
identification in accordance with the present invention. 
System 100 includes seven major subsystems: a ti 
mal imaging device 102, minutiae generator 104, a 
nutiaedata generator 105. minutiae matcher 106, m 
tiae database 111, microprocessor 110, and output 
play/processing device 1 1 4. In a preferred embodiment, 
minutiae generator 104, minutiae data generator 105, 
and minutiae matcher 106 are all implemented by pro- 
gram instructions stored in a program memory 108, and 
program memory 108, microprocessor 110, and minuti- 
ae database 111 are implemented by a programmed 
conventional computer 112. 

In operation, thermal imaging device 102 obtains a 
thermal image of the face of person 101 . A digital signal 
representative of the thermal image is provided as input 
to minutiae generator 1 04, which generates signals rep- 
resentative of thermal facial minutiae points for 101. 
These minutiae points have a number of characteristics 
including a specific location within the person or relative 
to other minutia, the apparent temperature at a given 
time, the temperature signature over a period of time, 
whether the minutia corresponds to a vein or artery, the 
width of the blood vessel, and the vector direction of 
branching blood vessels from the minutia These and 



other characteristics are sensed and data relative there- 
to are generated by a minutia data generator 105. This 
data is stored in the minutiae database 11 1. The minu- 
tiae matcher 106 compares minutiae data for known in- 
dividuals which has been stored in the database with 
that for unknown individuals currently being imaged by 
the imaging device 102. If a match is detected, a corre- 
sponding signal is sent to the output display/processing 
device 114. 

In a preferred embodiment, output display/process- 
ing device 114 comprises circuitry to permit or deny ac- 
cess to a secured facility depending on the results of the 
matching performed by minutiae matcher 106. In one 
embodiment, access is permitted if the person 101 is 
recognized as one of a group of authorized personnel. 
In a second embodiment, access is denied if the person 
1 01 is recognized as one of a group of unauthorized per- 
sonnel. In yet another embodiment, access is denied if 
the person 101 is not recognized by system 100. 
20 System 100 thus considers hidden micro parameter 
which lie below the skin surface, and which cannot be 
easily forged, if at all. The large number of such micro 
parameters considered renders it essentially impossible 
to search for a person to match another person's set of 
25 micro parameters. Furthermore, the particular infrared 
band used for imaging by Ihermal imaging device 102 
may be kept secret, or multiple bands may be used, 
which further increases the difficulty involved in compro- 
mising system 100 The underlying features detected by 
30 system 100 are essentially "hardwired" into the face at 
birth and remain relatively unaffected byaging, thus pro- 
viding for less inherent variability, than found in known 
recognition systems. Although thermal facial minutiae 
have some aspects related to, and extraclable from, ei- 
3S emental shapes and may be tagged to reflect the ele- 
mental shape parameters (such as by tagging with frac- 
tal dimensions), minutiae extraction does not require 
production or consideration of elemental shapes. Fur- 
thermore, the comparison ol thermal facial minutiae is 
40 computationally straightformard and introduces signifi- 
cantly less processing overhead than the known ap- 
proaches used for template or shape comparisons. 

Thermal imaging device 1 02 may be any device that 
produces a signal representative of the thermal charac- 
■« teristics of the face of person 101 . In a preferred embod- 
iment, a conventional digital video camera sensitive to 
thermal energy is used for the thermal imaging device 
102. As described herein, it is found thai tractable im- 
agery for facial identification may be derived from pas- 
so sively obtained infrared images of facial heat emana- 
tions which can be detected by commercially available 
thermal imaging devices sensitive in the 3 to 12 micron 
wavelength band. Unlike fingerprints Ihnt *re character- 
ized by a limited range of intensity values coi responding 
55 to three dimensional ridges which arc essentially con- 
centric rings about a single center, plus n^lous arch- 
es, line endings, and bifurcations, faci.v -■oimograms 
are generally characterized by conin- .-..-jiy varying 
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erator 104 next determines a vertical central line 208 
perpendicular to eye line 206 and mouth area 205 and 
intersecting eye line 206 midway between left and right 
canthi centroids 201, 202. Minutiae generator 104 then 
determines a lace center point 209 on central line 208 
midway between the points of intersection of vertical 
central line 208 with eye line 206 and mouth 205. Minu- 
tiae generator 104 further determines a horizontal cent- 
er line 210 perpendicular to the vertical centra! line 208 
and passing through face center point 209. Vertical cen- 
tral line 208 and horizontal central line 210 are desig- 
nated as face axes. Numerous other features may be 
. used to define face axes, but in general it is preferable 
to define face axes based on areas of the face that are 
not greatly deformable 

Other techniques may be used for location of the 
face center point 209 in those cases where the preferred 
use of facial symmetry and recognizable thermal fea- 
tures does not suffice. For example, other techniques 
may be called for with respect to facial images in which 
an eye patch is worn, eyeglasses are not symmetrical, 
only a partial face is imaged, the lower face is covered 
or the thermal pattern of the face is unusually distorted. 
The face center point 209 may in fact be outside of the 
boundaries of the facial image, for instance where only 
a partial facial image is obtained due to the face being 
partially blocked by another face or some other object. 
If the person 101 is wearing glasses, the pattern of the 
glasses, which typically block the infrared emissions 
from the face and thereby produce an extended cold ar- 30 
ea with sharp thermal discontinuity, can be used to de- 
termine approximate face axes. Additional techniques 
include manual location of the face center point 209 and 
preprocessing using known techniques to locate the ap- 
proximate area of the face center point 209. As de- 3S 
scribed below the face axes may be tested for validity 
to determine whether the image requires any such spe- 
cial treatment. 

II. Testing the Validity of Face Axes 40 

Since the known techniques for identifying left and 
right canthi centroids 201, 202, left and right nostrils 
203, 204, and mouth area 205 are subject to artifacts 
and other sources of error, and since some images of *5 
faces are significantly asymmetric or have features that 
are entirely missing (e.g., due to person 101 wearing an 
eye patch or having a disfigured face), minutiae due to 
person 101 wearing an eye patch or having a disfigured 



■ not met the face is considered to be a special case call- 
ing for manual intervention to determine the best ap- 
proximation for face axes. 

5 III. Normalization 

In practice, it is found that preprocessing through 
normalization of image size provides advantages in later 
recognition. Accordingly, minutiae generator 104 uses 
'0 the distances between left and right canthi centroids 201 
and 202 and the distance from face center 209 to eye 
line 206 to compare the size of facial thermogram 200 
with a standard image size. In a preferred embodiment, 
linear correction in the vertical and horizontal dimen- 
'5 sions is used to normalize the size of facial thermogram 
200 to match the standard, but other normalization mod- 
els could be used as well. 

IV. Production of Thermal Contour Lines 

?o 

As provided by thermal imaging device 102, facial 
thermogram 200 consists only of an ordered list of ther- 
mal values corresponding to each small portion of the 
imaged face. Minutiae generator 104 employs the fol- 
'5 lowing procedure to produce thermal contour lines for 
facial thermogram 200: 

a. For a digitized image having N bits of resolution, 
or 2 N bands of thermal values, determine thermal 

o contour lines having a particular "current" one of the 
2 N values. 

b. Produce minutiae in accordance with the steps 
below for the contour lines of the current value. 

5 

c. Repeat a and b above, each time using new one 
of the 2 N values for the "current" value, until the de- 
sired number of minutiae have been extracted of all 
of the possible values have been processed. 

} 

d. If the desired number of minutiae have not been 
extracted, repeat the process beginning with 2 N ' 1 
bands of values, and reduce the number of bands 
by 1 with each iteration, skipping those that are 
powers of 2, until the desired number of minutiae 
have been extracted or until no further reduction in 
bands can be achieved. 

Various other techniques for generating contour 



face), minutiae generator 1 04 performs checks to help so lines may also be used, with the goal being obtaining a 



spot instances where these points may have been 
correctly located or where unusual facial images are en- 
countered. First, a check is made to ensure that vertical 
central line 208 and mouth line 207 intersect within 
mouth area 205. Next, a check is made to ensure that 
vertical central line 208 intersects a line connecting left 
and right nostrils 203, 204 at point between left nostril 
203 and right nostril 204. If either of these conditions is 



sufficiently large number of minutiae for unique recog- 
nition, without producing too many spurious minutiae. 
Spurious minutiae increase processing overhead with- 
out benefitting recognition. The number of thermal 
bands that will produce an appropriate number of minu- 
tiae is readily determined by trial and error for any par- 
ticular application of system 100. 
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vides an additional advantage of compressing the data 
used for recognition. 

The wide variety of techniques for generating minu- 
tiae described above provides an added measure of se- 
curity, as one attempting to mimic thermal facial minuti- 
ae may be able to do so if one technique for generating 
minutiae is used by minutiae generator 104, but not if 
another is used. Thus, without prior knowledge of the 
particular technique being employed by minutiae gen- 
erator 104, system 100 becomes even more difficult to 
comprise than it otherwise might have been. 

As mentioned above, it may be desirable that all 
. thermal images be scaled to a standard size prior to 
processing. It also may be desirable, depending on the 
thermal imaging system used, that all thermal images 
first be normalized to a standard thermal profile before 
processing. In alternate embodiments, intended for var- 
ious applications and various environments, these pre- 
processing steps may significantly increase accuracy in 
recognition or may merely impose unnecessary 
processing overhead. For example, if system 100 is 
used in connection with an outdoor automated teller ma- 
chine, thermal normalization may be needed to deal with 
seasonally wide variations in surface skin temperature. 

Minutiae Matcher 106 

As mentioned above, minutiae generator 104 and 
minutiae data generator 1 05 are used to produce minu- 
tiae data signals for a population of known persons. The . 
data corresponding to these signals are stored in minu- 
tiae database 112. Thermal imaging device 102 then ob- 
tains a thermal image of an unknown person 101 and 
minutiae generator 104 produces signals representative 
of the minutiae and minutiae data generator 105 gener- 
ates data for the minutia for that person. Once these sig- 
nals have been produced, minutiae matcher 106 com- 
pares the signals representative of person 101 to sig- 
nals from minutiae database 102 corresponding to mi- 
nutiae data of known persons In a preferred embodi- 
ment, minutiae matcher 106 performs three basic func- 
tions to obtain a match: alignment of the unknown face, 
comparison of minutiae data, and selection of a match. 
Each of these functions is described in greater detail be- 
low. 

I. Alignment of Unknown Face 

Because there may not be control over the position 
of the face of person 1 01 with respect to the field of view i 
of thermal imaging device 102 when image is obtained, 
the orientation of the face may not be such that the facial 
axes are aligned to be horizontal and vertical. Thus, mi- 
nutiae matcher 106 corrects the orientation by rotating 
the image such that the facial axes are horizontal and - 
vertical. Next, conventional processing using a three di- 
mensional model is applied to correct for any rotation or 
twist of the head. In a preferred embodiment, such 



processing models the head as a sphere with a diameter 
equal to the apparent width of the face, and anti-distorts 
the image to provide a view which is normal to a surface 
plane across the forehead and upper lip and in which 
s the center of the sphere coincides with the face center. 
In a conventional manner, the nose and chin are ignored 
so as not to disrupt positioning of this surface plane. 

II. Comparison x>1 Minutiae 

Comparison of the minutiae data of the unknown 
person 101 with minutiae data from known persons be- 
gins by comparing locations of such minutiae. First, the 
locations of minutiae for a known face are considered, 
15 and denoted as M(K)i. Next an allowed positional error 
e is selected, as is determined to be appropriate for any 
given environment in which system 100 is used. The mi- 
nutiae of the known face are then overlaid on the minu- 
tiae of the unknown face, denoted M(U)j . Any M(U)j that 
>o are not with e of one of the M(K)i are ignored. Any M(K) 
i which are not within e of on the M(U)j are ignored. This 
leaves a residual set of minutiae pairs. Ifthis set is empty, 
there is not a match between the two images. Otherwise, 
the characteristics of the corresponding points are com- 
^5 pared. 

Depending on the application, any comparison 
technique that considers the characteristics (x, y, z, a, 
R, B, T) listed above may be used to generate a com- 
parison metric. In a preferred embodiment, only the po- 
30 sitional differences are considered 

The simplest decision technique is to set a minimum 
number of pairs of corresponding minutiae for a poten- 
tial match. If an unknown face and a known face exhibit 
a least the minimum number of corresponding minutiae 
35 pairs, they are considered to be a potential match. 

In an alternative embodiment, the Ax and ay values 
for each pair of corresponding minutiae are determined, 
and the distribution of Ay with respect to Ax is then de- 
termined for the overall set of minutiae pairs. The stand- 
ee ard deviation of that distribution is then compared 
• against a threshold standard deviation to determine 
whether a potential match exists. 

In still another technique, a new error measure e' is 
introduced dependent not only on location but on ther- 
ms mal value (z). Minutiae pairs are only considered if they, 
are within a certain thermal value difference Az as well 
as have locations within the distance error e, thereby 
satisfying new error measure e'. 

Further levels of decision requirements can similar- 
>o ly be added to produce the desired level of confidence 
in the match for the application at hand. Each possible 
comparison of the unknown face with known faces is 
performed, and then the known images are rank-or- 
dered according to the goodness of fit (e.g. closeness 
•5 in metric) with the unknown face. 
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number of type A and type C cells. If a lace is divided in 
to 36 grid cells as illustrated in Fig. 6, classes could be 
designated as nAmC, where n is the number of type A 
cells m is the number of type C cells, n+m = 36 - p and 
p is the number of B cells. Using this arrangement, 1 260 
classifications are possible. Alternatively, ranges of val- 
ues can be considered to be within the same class. 

As a further refinement to such classification, the 
degree of bilateral symmetry in distribution of type A 
cells and type C cells could be considered. If the face is 
divided into four quadrants designated upper right, low- 
er right, upper left, lower left, each quadrant having 9 
cells, a metric for classification could look at differences 
in the numbers of type A and type C cells in horizontally 
or vertically adjacent quadrants. Such metrics may be 
the absolute difference in minutiae between such quad- 
rant pairs, or may be simplified by merely indicating 
whether a left (or upper) quadrant has more, fewer, or 
equal minutiae as a corresponding right (or lower) quad- 
rant. 

Other possible classifications are based on geo- 
metric values of, and ratios among, the points and lines 
described in connection with Fig 4, once the face has 
been normalized as described above. In some applica- 
tions, a combination of visual and thermal attributes may 
be employed for classification For example, a ratio be- 
tween the distance between left and right canthi centro- 
ids 201, 202 in facial thermogram and the distance be- 
tween the left and right pupils as determined through 
visual imaging is found to be a useful metric for classi- 
fication, as is the ratio between the distance from eye 
line 206 to horizontal central line 210 and the distance 
from a line connecting the eyes to the tip of the nose as 
determined by visual imaging, as is the ratio between 
the distance between left and right nostrils 203, 204 and . 
the distance between the outer limits of the nostrils as 
determined by visual imaging. 

The usefulness of facial thermal imaging in recog- 
nition applications is increased by appropriately encod- 
ing thermal facial images so that consistent codes are < 
generated each time a facial thermogram of a person is 
obtained. Such a coding scheme reduces database 
search and minutiae matching overhead, thereby allow- 
ing faster processing using less expensive equipment. 
In a preferred embodiment, overlaying a grid on a lace i 
such that 144 cells cover the area of the face, and as- 
signing a binary code to each cell, such I hat the cell is 
encoded with a "1" if the cell contains one or more mi- 
nutiae and "0" if the cell does not contain any minutiae, 
is found in practice to yield good results. Since this en- s 
coding scheme preserves the relative location of each 
bit, it is straightforward to ignore selected bits in cases 
where only a portion of a face is imaged, due to obstruc- 
tion, disguise, or orientation. 

Use of such a "facecode" also facilitates straightfor-' Si 
ward verification and comparison techniques. In some 
verification applications, for example, a requirement that 
10% of the coded bits match may be considered suffi- 



cient to provide a desired level of confidence. Simple 
difference comparison on a bit-by-bit basis, which is 
computationally extremely efficient, is sufficient to de- 
termine the number of corresponding bits between a 

s code of an unknown face and that of a known face. 
Where multiple known faces exceed a threshold level of 
similarity, the one with the greater number of common 
bits is readily selected as a best match. 

Although the discussion above has been directed 

10 [o thermal images of faces, it should be recognized that 
similar techniques and systems may readily be applied 
to images of other body parts in accordance with the 
present invention. It should also be recognized that nu- 
merous other imaging modalities besides thermal imag- 

is ing may be employed in accordance with the present 
invention, for example x-ray, NMR, MRI, and CAT scan 
imaging. It should also be recognized that known 
schemes for pattern recognition and graph matching 
may be applied readily in accordance with the present 
20 invention, depending on the needs of a particular appli- 
cation. 

Standardized Infrared Minutiae Co-ordinate System 
(SIMCOS) " " ~ 

2S 

The method and apparatus described above for fa- 
cial minutiae extraction can be used to develop a stand- 
ardized minutiae co-ordinate system for identification of 
medical patients and for diagnosis of medical condi- 
30 lions. Because an infrared camera operates at a dis- 
tance from the patient and detects and records only ra- 
diant heat spontaneously emitted from the body surface, 
it constitutes a painless, non-invasive, passive method 
of recording patterns of body surface temperatures. 
« These patterns have been found to depend upon the 
underlying vascular structure and are unique for each 
person. Infrared identification therefore provides a 
method for uniquely identifying individuals under all 
lighting conditions, including total darkness. It is not 
o prone to forgery or multiple identity deception and so 
provides convenient and highly secure identification of 
individuals. The method for generating repeatable reg- 
istration points on the skin surface of the human body 
utilizes discrete minutiae points obtained from the ther- 
5 mal images. Visual characteristics of the body, such as 
size and shape and relative position of body parts, are 
maintained in the infrared image. In addition, the details 
of the vascular system are indicated by the distribution 
of temperature across the skin surface. Current infrared 
' cameras are sufficiently sensitive to temperature varia- 
tions that they clearly distinguish the skin directly over- 
laying blood vessels due to the thermal difference 
caused by the flow of warm blood. The vascular struc- 
ture appears as a white (hot) overlay of the circulatory 
; structure on top of a grey scale image of the thermal 
map of the body, as shown in Fig. 7. 

In Fig. 8, there is shown apparatus 100a for 
processing infrared images to yield repeatable minutiae 
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cially-available thermal imaging devices sensitive in the 
3 lo 5 or 8 lo 12 micron wavelength bands. Images are 
this type are shown in Figs. 7. 10. 12. and 13. 

Current infrared cameras produce a standard ana- 
log or digital output providing 30 frames per minule as 
shown in Figs. 7-10. Tracking the minutiae from frame 
to frame assisls in the exploitation of the dynamic IR im- 
agery by allowing measurements to be made over time 
from the same body locations while accommodating 
changes in position due to respiration, voluntary or in- 
voluntary movements of the subject, and intentional or 
accidental variation in the position of the imaging sys- 
tem. The use of infrared video imagery also allows the 
imagery to be recorded in real time for later analysis 
and provides a self -documenting chain of custody iden- 
tification of the person recorded, all without the neces- 
sity for the cooperation of the person being imaged 

Infrared imaging can be used to locate minuliae 
points over the entire body surface which correspond lo 
intersection points and branch points of the underlaying 
blood vessels This provides a built-in set of registration 
points on the body's surface which can be annotated 
onto images produced by any medical sensor used in 
conjunction with the thermal imager. The registration 
points then can be used to compare and combine med- 
ical images taken with different equipment at different 
times and under different conditions, facilitating compar- 
ison ol those images Also, the minutiae points provide 
reference points for continuous re-alignment of surgical 
instruments, radiation sources and other diagnostic or 
treatment equipment. Since the infrared camera is to- 
tally passive, it can be used continuously during other 
medical procedures to overlay precise registration 
points on the other images while also monitoring for 
overheating, shock hypothermia, renal failure, and oth- 
er medical conditions. At the same time, the pattern of 
minutiae points superimposed on each image provides 
positive identification of the patient Such applications 
are of particular importance during telemedicme proce- 



lying blood vessels. 

The technique for Ihormal minuliae extraction and 
matching can be summarised as follows: 

1. Current thermal image is digitized. 

2. Current image is divided into pixels, where the 
size ol the pixel relates lo the resolution or quality 
of the result desired 

3. Certain pixels are selected as minutiae points. 

4. Each minutia is assigned a vector having magni- 
tude and directional information in relation to the 

'5 surrounding characteristics of the thermal image 
Additional characteristics, such as type of minutia 
may also be recorded for each. Typically for each 
whole body thermal image, there would be on the 
order of 1200 minutiae 

20 

5. Set of minutiae vectors of the current image are 
compared by computer to the set of vectors of other 
images. 

25 6. Comparison results are used to determine corre- 
sponding minutiae from the two images and to 
morph or mathematically adjust one image wilh re- 
spect to the other lo facilitate comparison. 

3 o 7. Differences between the current image and da- 
tabase images are computed for either the entire 
image or for areas ol interest. 



The normal body is basically thermally bilaterally 
symmetric Side to side variations are typically less than 
0.25 degrees Celsius. This fact is used in assigning axes 
to the body's image Where the skin surface is unbroken 
there is gradual variation of temperatures across blood 
vessels, with the highest temperatures across the body 
surface being directly on top of major blood vessels. Ma- 
jor thermal discontinuities occur al entrances to body 
cavities such as the eye sockets, nostrils, or moth 
These provide global reference points lor automatic ori- 
entation of the thermal image Localand relatively minor 
discontinuities in the skin surface occur at scars, moles 
burns, and areas of infection. The thermal surface can 
be distorted through pressures and aclivilies such as 
eating, exercising, wearing tight hats and other clothing, 
sinus inflammation, infection, wsighl gain and loss and 
body position However, the minutiae points remain con- 
stant with respect to their position relative lo the under- 



lt is desirable that all thermal images in a database 
35 be normalized to a standard thermal range and be 
scaled to a standard size during search and comparison 
procedures. Both normalization and scaling eliminate 
some minute amount of identifying characteristics of a 
particular person or his condition However, the stand- 
*° ardization procedures greatly aid in the exploitation of 
the database by reducing the need to calibrate every 
imaging sensor used to produce images which will be 
filed in, or compared to database images. For example, 
m accident triage wilh no accurate ground truth refer- 
45 ence in the scene and possibly use of inferior quality 
imagers, standardization to constant size and thermal 
range is appropriate in order to match against database 
Furthermore, standardization facilitates use of simulat- 
ed imagery lor telemedicine and telesurgery applica- 
so lions. For example, when incorporated into the military's 
automated battlefield medical pod, real time normalized 
thermal minutiae can be used to properly position injec- 
tions and application ol external pressure to stop bleed- 

55 In addition, standardizing database images facili- 
tates comparison of imagery during growth from child- 
hood taadullhood. compilation of medicnl liberies of im- 
ages from large number ol people. andauiomated com- 
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are boih already using electronic images as well as x- 
rays and medical records. Laparoscopic surgery is an 
electronic form of surgery. Teleradiology. lelepathology. 
and teleconsulalion are already, widely accepted elec- 
tronic medical practices. 

As telemedicine and telesurgery become more 
common, there will be more potential for error in identi- 
fication of patients and the treatment to be performed, 
and more need to document the precise medical history 
and treatment procedures performed by a given doctor 
on a given day. Filing, recall and comparison of docu- 
mentation collected over time by different sensors at dif- 
ferent facilities will need to be automated to a greater 
degree, while protecting the privacy of the patients The 
identification technique of the invention offers a low- 
cosl. repeatable. non-invasive, passive system for 
standardization and registration of many current forms 
of medical imagery, while also offering an approach to 
high security maintenance of files with immediate ac- 
cess in emergency situations 

Identification of Drug and Alcohol Usage 

Many drugs, including cocaine and alcohol, are va- 
soconstrictive substances which cause cooling of the 
skin surface, the resultant cooling is detected through 
passive imaging of the thermal energy emitted form the 
lace. In Fig 1 5 is shown the thermal image of an individ- 
ual who is substance free and in Fig. 16 is a thermal 
image of the same individual after the ingestion of alco- 
hol. 

The thermal imaging techniques of the present in- 
vention can be used to detect substance use by individ- 
uals, even where the individual's identity is unknown. 
This is accomplished by compiling databases of statis- 
tical analysis of thermal signatures obtained from clini- 
cal trials in which cooperating subjects have concurrent 
drug testing performed using urinalysis or blood testing 
along with thermal signatures obtained from known sub- 
population but without concurrent testing by other 
means. 

The vascular system supplying the human face typ- 
ically exhibits thermal variations on the order of 7°C 
across the facial surface Certain general features such 
as hot patches in the sinus areas, relatively cool cheeks, 
and cold hair pertain to all facial thermograms. Other 
features such as specific thermal shapes in certain ar- 
eas of Ihe face are characteristic of a particular person. 
Measured disturbances to other features, such as Ihe 
general symmetry between two sides of face, range of 
thermal variations in Ihe forehead, peak temperalure, 
size of canthi pattern, and variations in those disturbanc- 
es over time, may be correlated with a high probability 
of drug or alcohol use. 

Variations in temperature across the facial surface 
can be imaged by thermal cameras sensitive to wave- 
lengths in the 3-5. 3-12. or 2-1 5 micron ranges. Current 
cameras can provide thermal resolution better than 



0.07°C and spalinl resolution of better than 02": result- 
ing in 65.000 lo 265.000 discrete thermal measure- 
ments across the surface of the face. For most such 
cameras that thermal map is regenerated 30 limes per 

5 second to produce either a standard video output which 
can then be recorded and processed on standard vide- 
otape equipment, or a direct digital signal which can be 
immediately input to a computer. 

Certain drugs appear to produce characteristics 

io leatures in facial thermograms, which may be identifia- 
ble from detailed analysis ol Ihe structural patterns and 
distribution statistics. Furthermore, the rate of change 
at any point in lime may be a discriminator between 
chronic and recent use of each drug. Using currently 

'5 available thermal imaging cameras, thermal signatures 
emitted from the face can be used to deduce changes 
in activity levels of specific arteries in (he brain which 
are known to be affecled by particular drugs. 

The vascular system has a common structure in 

20 each person, with known pathways for instance (rom the 
heart lo Ihe brain and known pathways between blood 
vessels in the face and those in the brain as shown in 
Figs. 2b and 2c. Using the SIMCOS technique, a set of 
standardized minutiae appearing in the face can be 

25 identified. Through clinical drug trials using known 
types amounts, purity, and administration techniques. 
Ihe thermal eflecls over lime at each such minutiae lo- 
cation can be observed The effect ot varying the type, 
amount, or purity of drug can also be observed. The ef- 

30 lect on different people can be observed. Since the ther- 
mal effects may be quite small and localized, it is impor- 
tant to utilize Ihe SIMCOS method lor identifying the pre- 
cise minutiae locations in each subject. Thai provides 
repeatability of measurements over time without requir- 
es ing the application of registration markers to the face, or 
the use of invasive techniques to repeatably find Ihe 
same locations. Also, it provides a method for compar- 
ing corresponding locations in different subjects. 

Statistical analysis of time-varying thermal signa- 

■10 tures at each lacial minuliae point before during and 
alter drug or alcohol administration provides a reference 
dataset which represents the thermal eflect of that sub- 
stance under the protocol used. A library ol thermal mi- 
nutiae substance effect signatures can be developed for 

j s various drugs and other substances for which screening 
is desired, including for prescription and over the coun- 
ter medications, tobacco, and alcohol. Fig. 17 presents 
an illustration ol thermal signatures associated with sub- 
stance-free subjects. These may differentiate between 

so sex. age, size, medical history, or other characteristics 
ol the substance-free subjects. In addition, thermal mi- 
nuliae on non-substance signatures may be developed 
lor each person enrolled in to a system who will be sub- 
sequently scanned lor substance use. Al the lime that 

ss the thermal data is collected, urinalysis or blood lestmg 
can be performed to assure that the sub|oct is sub- 
stance -free 

Subsequently, each time a person ..^tolled i' 1 lno 
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proach involves sampling the thermal waveforms and 
producing a matrix of values where one dimension of 
the matrix is the number of minutiae used, and the other 
is the number of temperature samples over lime. The 
reference library can include wider malrices. involving 
longer time period than is practical for an operational 
screening system. The comparison between the collect- 
ed matrix and the reference matrix would use a digital 
shifting and difference calculation to find the best area 
of match. 

A measure of goodness of match is made between 
the collected thermal signatures and the signatures for 
each substance under each protocol in the library. The 
system manager selects a threshold to be applied to 
each comparison, such that matches which are closer 
than that threshold will cause the system to issue a no- 
tice of possible substance detection 

The results of comparison with the different markers 
may be recorded or stored or output to decision mark- 



suitable size and position. If the image is not suitable 
according to software criteria established within the face 
locator, a new image frame is grabbed and the process 
repeated until a suitable image is obtained. 

The image is then processed by the minutiae ex- 
tractor and identifier 22 which locates the SIMCOS mi- 
nutiae points and extracts their positions on the image 
and the corresponding apparent temperatures. Addi- 
tional frames are grabbed and processed for a period of 
time selected by the system manager. 

The extracted minutiae locations and correspond- 
ing temperatures are processed by the thermal signa- 
ture extractor 24 which generates for each minutiae 
point the thermal variation over lime For enrollment pur- 
poses, a single frame may suffice. However, multiple 
frames over a period of seconds should be taken in or- 
der to help calibrate and factor-oul noise in the system. 
Personal data about the enrollee is transferred to 



..... , ^ , , 'he enrollment database 26, along with the thermal sig- 

nLZT T\ ^ ldsma y bea "^»yaP- *> natures extracted for the enrollee If the enrollee I 

nlieri tn Ihp rain iiat^H Hiff-^,^^^^^ r _ «. .. , , c lo 

known to be substance-free, the thermal signatures arp> 



plied to the calculated differences to render a pass/fail 
or clean/under-influence determination. The statistical 
estimate of confidence in the determination can also be 
presented 

The apparatus lor drug and alcohol detection is 
shown in Fig. 1 8. Three primary functions are performed 
with the apparatus; Enrollment, Reference Signatures 
Development, and Screening. System components may 
in general be rendered as software, hardware, or 
firmware elements. 

Prior to automated operation of the Identification 
and Detection (ID&D) system, a human operator termed 
a System Manager must perform set-up and initializa- 
tion of the system, which he does via the System Man- 
ager Interlace, which includes a monitor keyboard, and 
possibly a printer and other peripherals such as a mouse 
which are normally associated with personal computers. 
A System Manager must confirm the identity of the en- 
rollee 4, and input (he associated identification informa- 
tion into the person identifier and condition identifier da- 
tabase 6 within the system processor 8. The enrollee's 
current and past medical history data is also input to the 
system, including the results of urinalysis or blood tests 
todetecl substance use, current use of illicit substances, 
and other information which may bear upon alcohol and 
drug testing results. 

The enrollee stands or sits at designated location 
facing the infrared camera 12 and within reach of the 
event trigger 14. When the event trigger is engaged by 
either the enrollee or System Manager, the output from 
the infrared camera is sampled by the frame grabber 1 6 
and the resulting frame stored in its buffer At the same 
time the camera output is recorded on a video cassette 
recorder 18 which incorporates annotation of the date 
time, location, and identify of the enrollee. 

The image in the frame grabber and buffer is proc- 
essed by the face locator 20 which determines lhal the 
image includes a single face which is in locus nnd ol a 



also transferred to the substance-free signature data- 
base 28 stored by enrollee and also to the substance- 
free thermal signature database stored by classification 
?s of enrollee 30. 

If the enrollee is known or found to be substance- 
influenced, the thermal signatures are instead trans- 
ferred to the substance-related thermal signature data- 
base stored by substance and also to the substance- 
'0 related thermal signature database stored by protocol 
34. The definitions of protocols will relate to clinical trial 
used for developing reference thermal signatures, and 
may also include self-reporting classifications such as 
"heavy" regular user of cocaine", or "infrequent user of 
5 marijuana but not within the past month 

The extracled thermal signatures of the enrollee are 
also transferred to the thermal signaluio comparator 
and statistical analyser 36 which compi-nes the signa- 
tures of the enrollee with other signalmen in the data- 
' bases. If the enrollee's signatures vary too much from 
the others in the same substance-free cUii or from oth- 
ers in the same substance-related or piotocol-related 
databases, then the system manager m-,y icquest re- 
view by the medical review officer 38 pn-;: to including 
• the enrollee's data in the database. 

If no anomaly is detected in the cm - :-c's thermal 
signature, then the enrollee is instruct.:.: :-, to how to 
activate the system for future access »r : ... -i coning. If 
a personal identification number is to b-~ . ri that PIN 
will be assigned. If voice recognition if : • or other 
technique for. identification is to be us-- : >c proce- 
dures will be taught. The enrollee is no • -od in the 
system. 

The substance use identification r-.- :on sys- 

tem requires databases of thermal sic:- ■ ::n sub- 

stance-free and substance-relaled in •• j clin- 

ical trials in which substances are h : mder 

rigorous protocols the same appnor to 
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grant access Ihrough manual intervention or through au- 
tomatic control of an access portal. Depending upon the 
particular classification selected, or the amount of vari- 
ation between the entrant and the selected reference 
signatures the result classification may be sent cither 
automatically or through manual intervention to a med- 
ical review officer 40 for a final determination. The en- 
trant's file from the enrollment database 26 is also sent 
to the MRO to provide background information. 

Statistical analysis of drug or alcohol use within a 
random population is also encompassed by the present 
invention. This is performed by scanning a crowd and 
locating faces therein for analysis. 

Various standard methods lor locating laces in an 
image frame can be used A particularly useful approach 
uses an ellipse detector to find relatively warm ellipses 
(thermal laces) within a relatively cool background. The 
ellipse is located within the expected height range for 
humans, the delected temperature is within the expect- 
ed range for human faces, and characteristics common 
to all facial thermograms (hot canthi regions symmetry 
of the eyes, nostrils, ears cheeks, etc.) are not violated. 
Each detected (ace in each frame is evaluated to deter- 
mine if it meets the quality requirements lor further 
processing. Requirements include the facial image be- 
ing in focus, being large enough to provide adequate 
resolution of the facial minutiae, being oriented close 
enough to full lace lorward and being free enough of 
blockages including beards, eyeglasses, and interven- 
ing obstructions, such that a sufficient number ol facial 
minutiae can be extracted from the facial image The 
specific requirements are dependent upon what sub- 
stances are to be detected, in how small a dose, and 
after what period of time. Faces which do not meet the 
quality requirement are not further considered. Those 
which are qualified are assigned unique tags. 

The next frame is then processed and quality faces 
are detected as above. Each such face is Ihen com- 
pared to the faces in the previous Irame. or to those fac- 
es from the previous frame which are close enough in 
location that they could be a particular face in the current 
frame Matching is performed using the facial minutiae 
matching method If a current and prior face are deter- 
mined to be the same, then they are given the same lag. 
This process continues with subsequent trames being 
likewise analyzed In general, a particular tagged face 
will move across and then out ol the lield ol view When 
the face is no longer seen by the camera then the ther- 
mal signatures associated with each of its minutiae in 
each of the frames are combined and matched against 
a reference database (or non-substance and sub- 
stance-related signatures 

The system is designed (or stand-alone operation 
IT is deployable for programmable periods ol lime, dur- 
ing which it will analyze and classify each lace which 
appears within its field of view. The system will not rou- 
tinely record or store the thermal images, although pro- 
visions are made todo that during testing and evaluation 



of the system in order to allow for improvements to be 
made in the system and compared with earlier results. 
The output from the system will be graphical results 
such as shown in Fig 19 The cumulative detection in- 

s dex of the y-axis represents the number ol persons who 
the system estimates have used marijuana, cocaine or 
heroin in an amount and within a timeframe which re- 
sults in a residual level indicated by the x-axis value at 
the time of the analysis. The x-axis represents the con- 

'0 fidence level of drug signature indication, which is relat- 
ed to the detection precision of the testing and analysis 
procedures Separate curves indicate the specific drugs 
delected, and a composite survey indicates detection of 
any of the substances. Due to the frequent use ol corn- 
's binations of drugs the composite curve is expected to 
be more significant than its components 

The system is tested using known populations ol 
drug users, and its results compared to urinalysis re- 
sults. The comparison is used to select thresholds for 

20 system decisions on classification of thermal signa- 
tures. The system can be deployed within a high inten- 
sity drug trafficking area and its results compared to 
other current estimates of drug usage in that area. 
The apparatus for statistical analysis is shown in 

25 Fig. 20. Three primary functions are performed with Ihe 
apparatus: face acquisition and tagging: face analysis 
and classification: and statistical population analysis. 

An infrared camera 42 is positioned such that per- 
sons in the population to be scanned generally enter. 

30 transverse, and exit the camera's field ol view 44. The 
infrared camera produces a sequence of frames using 
either direct digital output or Irame-grabbed video output 
46 which is stored in the frame bufler storage 48. 
Processing ol the imagery is initiated by an initializer 

35 mechanism 50 which may be a proximily detector mo- 
tion detector, or other sensor used to detect the possible 
presence of humans within the field of view. 

The system processor 52 is comprised of five com- 
ponents The face locator 54 applies rules to the buffer 

JO stored image to identify all faces in the frame The face 
quality check 22 applies additional rules lo determine if 
a given lace provides sufficient information in terms ol 
focus, resolution, position, and number of minutiae 
available. Each qualified lace is tagged and then com- 

-»s pared with qualified faces in the preceding frame, using 
the SIMCOS technique, to determine which faces have 
already been seen and tagged. 

Faces which are seen in a given frame may have 
been temporarily blocked in the previous frames, and 

so so any (ace may be blocked, turned, or otherwise un- 
qualified part of Ihe time Comparison against earlier 
qualified faces is continued for a period ol lime which is 
considered a reasonable maximum lime lor transvers- 
ing the filed of view. The face tracker 54 rcnssigns tags 

55 so that Ihe same (ace receives the same trig in subse- 
quent frames. There are two reasons lo bo concerned 
about consistently applying the same Inq to each face 
First, drug and alcohol detection accuracy improves 
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visual characteristics are thermal characteristics 

8. A system as dotined in claim 7. wherein said minu- 
tiae are limited to those in the face ol the individual 

5 

9. A method for classifying an individual, comprising 
the steps of 

(a) sensing characteristics of the individual: 

(b) producing a normalized representation of 
the individual in response to the sensed char- 
acteristics thereof: 

(c) identifying minutiae of the individual in re- >s 
sponse to the normalized representation said 
minutiae corresponding to specific branch 
points of blood vessels of the individual: 

(d) determining a correspondence between a 20 
grid of cells and the normalized representation 

and 

(e) classifying the individual in response toco- 
location of at least one of said minutiae and at 25 
least one of said cells, thereby identifying the 
individual as corresponding to a class. 

10. A method as defined in claim S, wherein said clas- 
sifying step includes encoding the identity of the in- 30 
dividual using a plurality of bits each corresponding 

to one of said cells, setting one of the plurality of 
bits to a first state in response to the presence in a 
corresponding cell of one of said minutiae and to a 
second state in response to the absence in the cor- 35 
responding cell of any of said minutiae 

11. A non-invasive method (or identifying medical pa- 
tients, comprising the steps of 

40 

(a) identifying minutiae on the body of a known 
patient, said minutiae corresponding with 
branch points of blood vessels: 

(b) storing a collection of minutiae data charac- is 
teristics of the minutiae for the known patient in 

a memory to define a reference collection, said 
reference collection being unique to the known 
patient: 

50 

(c) sampling the minutiae data of an unknown 
patient in a selected location of the unknown 
patient's body to define a sample collection: 

(d) comparing said sample collection with said 55 
reference collection whereby the identity of the 
unknown patient can be confirmed when s.-='Cl 
sample and reference collections correspond 



12. A method as defined in claim 11. and further com- 
prising the steps of 

(e) detecting any changes in said sample pat- 
tern with respect to said reference collection, 
and 

(f) analyzing the changes to diagnose the oc- 
currence of a medical event in the patient. 

13. A non-invasive method lor diagnosing a medical 
condition in an individual comprising the steps of 

(a) establishing a reference collection of series 
of time-varying minutiae data corresponding 
with a plurality of known medical conditions, re- 
spectively, said minutiae data being derived 
from characteristics of minutiae associated with 
branch points of blood vessels in a human be- 
ing: 

(b) generating time-varying minutiae data from 
the individual having an unknown medical con- 
dition: and 

(c) correlating said generated minuliae data 
with said reference minutiae data to diagnose 
the medical condition when a match between 
said generated and reference data is obtained. 

14. Apparatus for identifying the use of a substance 
such as alcohol and drugs by an individual compris- 
ing 

(a) means for identifying minutiae data in the 
lace of the individual which are responsive to 
the ingestion of a substance by the individual, 
said minutiae data being derived from charac- 
teristics of minutiae associated with branch 
points of blood vessels of the individual: 

(b) minutiae data generator means for generat; 
ing a first collection of minutiae data ol a sub- 
stance-free individual and a plurality of second 
collections of minutiae data of an individual af- 
ter the ingestion of a plurality of different known 
substances: 

(c) means for storing said first and second col- 
lections of minutiae data in a database: and 

(d) means for comparing a subsequent collec- 
tion of minutiae data of an individual with said 
first and second collections to determine 
whether the individual is substance dee and iq 
determine what substance the individual has in- 
vested where (he individual is not subslance- 
(ree. 
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responding therewith, respectively, whereby a 
plurality ol collections are generated, one lor 
each individual in the random population. 

25. A method lor determining what portion of a random 5 
population has ingested a substance such as drugs 
and alcohol, comprising the steps of 

(a) generating a plurality reference collections 

of minutiae data corresponding with substance- to 
free and substance-affected individuals : 

(b) generating current collections of minutiae 
data for individuals in the random population, 
respectively: and 'S 

(c) comparing said current collections of minu- 
tiae data with said reference collections ol mi- 
nutiae data in order to determine the portion of 
the random population that is substance-free 2<? 
and the portion of the random population that 

is substance-affected 

26. A method as defined in claim 25 wherein said ref- 
erence collection of minutiae data include different zs 
patterns corresponding with known ingested sub- 
stances, whereby the portion of Ihe random popu- 
lation which has ingested each known substance 
can be identified. 

27. A method for annotating an image of the human 
body comprising the steps of , 

(a) generating an image of the human body with 

an infrared camera: and 35 

(b) superimposing on said image the location 
of minutiae detected by said camera, said mi- 
nutiae corresponding to specific branch points 

of blood vessels of the body -to 

28. A method as defined in claim 27, and further com- 
prising the step of identifying particular minutiae to 
serve as reference points 

■is 

29. A method as defined in claim 27. and further com- 
prising the steps ol generating a medical image of 
Ihe human body and annotating the medical image 
with said minutiae. 

so 

30. A method as defined in claim 29, and further com- 
prising Ihe step of identifying particular minutiae to 
serve as reference points. 

3t. Apparatus for annotating an image of the human ss 
body, comprising 

(a) means for generating an infrared image of 



the body: 

(b) means for generating a pattern ol minutiae 
Irom said infrared image, said minutiae corre- 
sponding with specific branch points of blood 
vessels ol the body. 

(c) .means for generating a medical image ol the 
body: and 

(d) means for annotating said pattern of minu- 
tiae on to said medical image. 

32. A method lor maintaining the position of a surgical 
instrument relative to a surgical site during a surgi- 
cal procedure, comprising the steps of 

(a) identifying minutiae in the vicinity of a sur- 
gical site on a patient, said minutiae corre- 
sponding with specific branch points of blood 
vessels of the patient: 

(b) generating a reference minutiae pattern for 
a stationary patient: 

(c) generating a reference position of a surgical 
instrument with respect to said reference pat- 

(d) detecting deviations from said reference 
pattern owing to movement of the patient and 
from said instrument reference position owing 
to displacement ofthe instrument: and 

(e) repositioning the instrument with respect to 
said reference pattern to accurately position 
the instrument at the surgical site. 

33. Apparatus for maintaining the position of a surgical 
instrument relative to a surgical site during a surgi- 
cal procedure, comprising 

(a) means for identilying minutiae in the vicinity 
of a surgical site on a patient, said minutiae cor- 
responding with specific branch points of blood 
vessels of the patient: 

(b) means for generating a reference minutiae 
pattern for a stationary patient: 

(c) means for generating a reference position 
of a surgical instrument with respect to said ref- 
erence pattern: 

• (d) means for detecting deviations from said 
reference pattern owing to movement of Ihe pa- 
tient and Irqm said instrument icl'Mcnco posi- 
tion owing to displacement of Hie instrument. 
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(54) Method and apparatus for annotation of medical imagery to facilitate patient identification, 
diagnosis and treatment 

(57) A method and apparatus for annotation of med- 
ical imagery to facilitate patient identification, diagnosis, 
and treatment is characterized by an imaging device for 
producing a first signal representative of sensed char- 
acteristics of the individual and a minutiae generator 
which receives the first signal and produces a second 
signal representative of minutiae of the individual, the 
minutiae corresponding to specific branch points of 
blood vessels of the individual A minutiae data genera- 
tor analyzes the characteristics of minutiae and produc- 
es a third signal representative of the characteristics 
which is stored in a minutiae database for each of the 
plurality of known individuals and their medical condi- 
tions. The minutiae and minutiae data may be used to 
annotate medical imagery to facilitate subsequent im- 
age comparison by providing standardized registration 
points and time-varying characteristics. A minutiae 
matcher pairs corresponding second signals and third 
signals from a current patient with those from a data- 
base record, and the paired signals are used to align the 
images and compare them. The minutiae analysis tech- 
niques of the invention can be used to identify medical 
patients, assist in the diagnosis of medical conditions, 
and detect and monitor the use of alcohol and drugs, 
including anesthesia. 
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1. Claims: 1-12 

Classification or identification of persons based on 
features relating to bloodvessel patterns 



2. Claims: 13-26 

Diagnosing a condition of a person or persons based on 
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3. Claims: 27-33 

Using particular parts of a bloodvessel pattern as 
positional reference points 



